Predicting disease progression from tissue images and tissue segmentation maps

ABSTRACT

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating a final progression score characterizing a likelihood that a state of a medical condition affecting eye tissue will progress to a target state in a future interval of time. In one aspect, a method comprises: obtaining: (i) an input image of eye tissue captured using an imaging modality, and (ii) a segmentation map of the eye tissue in the input image into a plurality of tissue types; providing the input image to each of one or more first classification neural networks to obtain a respective first progression score from each first classification neural network; providing the segmentation map to each of one or more second classification neural networks to obtain a respective second progression score from each second classification neural network; and generating the final progression score based on the first and second progression scores.

BACKGROUND

This specification relates to processing data using machine learningmodels.

Machine learning models receive an input and generate an output, e.g., apredicted output, based on the received input. Some machine learningmodels are parametric models and generate the output based on thereceived input and on values of the parameters of the model.

Some machine learning models are deep models that employ multiple layersof models to generate an output for a received input. For example, adeep neural network is a deep machine learning model that includes anoutput layer and one or more hidden layers that each apply a non-lineartransformation to a received input to generate an output.

SUMMARY

This specification generally describes a system implemented as computerprograms on one or more computers in one or more locations thatprocesses a tissue image depicting tissue in a region of the body of apatient to generate a progression score for a medical conditionaffecting the tissue. Throughout this specification, a “progressionscore” may refer to a numerical value that characterizes a likelihoodthat the state of the medical condition affecting the tissue willprogress to a target state in a future interval of time. The futureinterval of time may be, e.g., an interval of time having a specifiedduration starting from a “current” time point, e.g., the time point whenthe tissue image was captured.

According to a first aspect there is provided a system comprising one ormore computers and one or more storage devices storing instructions thatwhen executed by the one or more computers cause the one or morecomputers to implement one or more first classification neural networks,one or more second classification neural networks, and a subsystem.

Each first classification neural network is configured to receive animage of eye tissue captured using an imaging modality, and process theimage to generate a first progression score characterizing a likelihoodthat a state of a medical condition affecting the eye tissue willprogress to a target state in a future interval of time. Each secondclassification neural network is configured to receive a segmentationmap of an image of eye tissue that segments the eye tissue in the imageinto a plurality of tissue types, and process the segmentation map togenerate a second progression score characterizing a likelihood that astate of a medical condition affecting the eye tissue will progress to atarget state in a future interval of time. The subsystem is configuredto obtain: (i) an input image of eye tissue captured using an imagingmodality, and (ii) a segmentation map of the eye tissue in the inputimage into a plurality of tissue types, and generate, based on the inputimage and the segmentation map, a final progression score characterizinga likelihood that a state of a medical condition affecting the eyetissue will progress to a target state in a future interval of time, byproviding the input image to each of the first classification neuralnetworks to obtain a respective first progression score from each firstclassification neural network, providing the segmentation map to each ofthe second classification neural networks to obtain a respective secondprogression score from each second classification neural network, andgenerating the final progression score based on the first progressionscores and the second progression scores.

Some advantages of this approach are described later, but in broad termsit has been found that the first and second classification neuralnetworks, one working on a “raw” image of the eye tissue, the otherworking on a tissue segmentation map, complement one another'sperformance when processing representations of eye tissue to identifywhen a medical condition will progress.

As described, a likelihood of progression of the medical condition ischaracterized by the final progression score. Looked at differently thefinal progression score may be considered as evaluating a condition ofthe eye tissue, in particular where the evaluation determines whetherthe eye has pathology which indicates that progression to the targetstate is likely (e.g. above a treatment threshold) within the futureinterval of time. The system may determine, based on the finalprogression score, that treatment, e.g. preventative treatment, shouldbe administered to the eye tissue.

In implementations of the system the medical condition affecting the eyetissue is age-related macular degeneration (AMD), in particular dry AMD.In implementations the target state of the medical condition affectingthe eye tissue is neovascular age-related macular degeneration (nAMD),also referred to as exudative AMD (exAMD) or wet AMD. In someapplications the medical condition is (dry) AMD in the eye of a patientwhere the patient's other (fellow) eye has been diagnosed as having wetAMD (i.e. nAMD or exAMD) and the target state is wet AMD.

The image of the eye tissue captured and processed by the firstclassification neural network(s) may be an optical coherence tomography(OCT) image, but this is not essential and the image of the eye tissuemay be captured using other techniques. The image may, but need not be,be a three-dimensional image comprising a plurality of voxels.

The segmentation map may be generated manually but in implementationsthe system may include one or more segmentation neural networks. Eachsegmentation neural network may be configured to receive an image of eyetissue captured using the imaging modality, and process the image togenerate a segmentation map of the image that segments the eye tissue inthe image into a plurality of tissue types. Where there is more than onesegmentation neural network the subsystem may be configured to providethe input image to each of the segmentation neural networks to obtain arespective initial segmentation map of the eye tissue in the input imageinto the plurality of tissue types from each segmentation neuralnetwork, and generate the segmentation map based on the initialsegmentation maps e.g. by averaging.

In implementations the first and second classification neural network(s)may be trained to generate additional outputs characterizing referraldecisions and additional diagnoses. Such outputs may be used whentraining the system and afterwards disregarded. A referral decision maybe a decision (of a clinician) defining a need for further medicalattention for a patient with the medical condition e.g. specifying anurgency with which the patient should receive further medical attention.An additional diagnosis may be a diagnosis of a pathology of the imagedeye in addition to the medical condition, e.g. in addition to AMD.

In some implementations, each segmentation neural network is aconvolutional neural network having a U-Net architecture.

In some implementations, the input image of eye tissue captured usingthe imaging modality is a three-dimensional image comprising a pluralityof voxels, and the segmentation map assigns a respective tissue typefrom a predetermined set of tissue types to each of the voxels.

In some implementations, the predetermined set of tissue types compriseone or more anatomical tissue types and one or more pathological tissuetypes.

In some implementations, the future interval of time is an interval oftime starting from a current time point.

In some implementations, the system performs operations furthercomprising generating, for each of a plurality of given future intervalsof time, a respective final progression score characterizing alikelihood that the state of the medical condition affecting the eyetissue will progress to the target state in the given future interval oftime.

In some implementations, the system performs operations furthercomprising determining, based on the final progression score, thatpreventative treatment should be administered to the eye tissue.

In some implementations, the first classification neural networks andthe second classification neural networks are trained to generateadditional outputs characterizing referral decisions and additionaldiagnoses.

In some implementations, the system comprises a plurality of firstclassification neural networks, wherein each first classification neuralnetwork has a same architecture but has been trained (i) on a differentset of training data, (ii) with differently initialized parameters, or(iii) both, from each other first classification neural network.

In some implementations, the system comprises a plurality of secondclassification neural networks, wherein each second classificationneural network has a same architecture but has been trained (i) on adifferent set of training data, (ii) with differently initializedparameters, or (iii) both, from each other second classification neuralnetwork.

In some implementations, providing the input image to each of the firstclassification neural networks and providing the segmentation map toeach of the second classification neural networks comprises performingtest-time data augmentation.

In some implementations, generating the final progression score based onthe first progression scores and the second progression scores comprisesaveraging the first progression scores and the second progressionscores.

According to another aspect there is provided a computer-implementedmethod comprising the operations of the system described herein.

According to another aspect there are provided one or morenon-transitory computer storage media storing instructions that whenexecuted by one or more computers cause the one or more computers toperform the operations of the system described herein.

Particular embodiments of the subject matter described in thisspecification can be implemented so as to realize one or more of thefollowing advantages.

The system described in this specification can predict the progressionof a medical condition affecting a patient. For example, the system canbe used to predict whether an early, mild form of age-related maculardegeneration (AMD) affecting an eye of a patient will convert tosight-threatening neovascular AMD (nAMD) or exudative wet AMD (exAMD) ina 6-month time period. Therefore the system can be used to facilitateeffective provision of medical care, e.g., by identifying patients whoshould receive preventative medical treatments, or by identifyingpatients where regular follow-ups are necessary to closely monitorprogression of a medical condition.

As part of generating a progression score characterizing the predictedprogression of a medical condition in a patient, the system receives atissue (anatomical) image showing a region of the body of the patient,and generates a segmentation map that segments the tissue image intomultiple tissue classes. The system then processes the tissue image togenerate one or more respective progression scores, processes thesegmentation map to generate one or more respective progression scores,and ensembles (i.e., combines) all the generated progression scores togenerate a final progression score. The progression scores generatedbased on the tissue image and the progression scores generated based onthe segmentation map may be different and complementary. For example,the progression scores generated based on the tissue image may reflectimage features not captured by the segmentation map that may be relevantto the progression prediction task, e.g., patterns in the reflectivityof the tissue in an OCT image of an eye. The progression scoresgenerated based on the segmentation map may be more stable (i.e., lesslikely to assume inaccurate outlier values) than those generated usingthe tissue image, e.g., because of the potentially lower complexity ofthe segmentation map relative to the tissue image. Therefore, generatingthe final progression score based on both the tissue image and thesegmentation map may improve the performance, e.g., the accuracy androbustness, of the system.

The segmentation maps generated by the system may provide a user of thesystem with a clinically interpretable indication of some of theevidence used by the system in generating its predictions, which canenable the user to assess the reliability of the predictions. Incontrast, some conventional systems operate as “black boxes” that do notreveal any insight into how predictions are generated.

The system can generate a final progression score characterizing thepredicted progression of disease in a patient by combining a collectionof progression scores (in some cases, hundreds of progression scores)made by respective progression classification neural networks that havebeen trained differently, that process different inputs, or both. Thiscan enable the system to generate final progression scores that arestable and reliable, thereby making the system more appropriate fordeployment in clinical workflows. Moreover, since the progression scoresgenerated by the progression classification neural networks areaggregated, the system may train each progression classification neuralnetwork on a lesser amount of training data than might otherwise berequired while maintaining the overall accuracy of the system.Therefore, the system may use less memory to store the training dataused to train the progression classification neural networks, therebyreducing use of memory resources in comparison to some other systems.

The details of one or more embodiments of the subject matter of thisspecification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages of thesubject matter will become apparent from the description, the drawings,and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an example disease progression prediction system.

FIG. 2 illustrates predictions made by a disease progression predictionsystem by processing multiple OCT images of an eye of a patient over aperiod of months to generate progression scores characterizing alikelihood that the eye would progress from early AMD to nAMD within a6-month period.

FIG. 3 is a flow diagram of an example process for generating a finalprogression score characterizing a likelihood that the state of amedical condition affecting a patient will progress to a target state ina future interval of time.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

FIG. 1 shows an example disease progression prediction system 100. Thedisease progression prediction system 100 is an example of a systemimplemented as computer programs on one or more computers in one or morelocations in which the systems, components, and techniques describedbelow are implemented.

The system 100 is configured to process a tissue image 102 depictingtissue in a region of the body of a patient 104 to generate a respective“final” progression score 106 for each of one or more future intervalsof time. The final progression score 106 for a future interval of timecharacterizes a likelihood that the state of a medical conditionaffecting the tissue depicted in the tissue image 102 will progress to atarget state in the future interval of time. Each future interval oftime may be, e.g., an interval of time having a specified durationstarting from a “current” time point, e.g., the time point when thetissue image 102 was captured. In one example, the system 100 maygenerate a respective final progression score 106 for future intervalsof time having durations 3 months, 6 months, 9 months, and 12 months,starting from a current time point.

The tissue image 102 may be a two-dimensional (2D) image (e.g.,represented as a 2D array of pixels), a three dimensional (3D) image(e.g., represented as a 3D array of voxels), or a higher dimensionalimage. The tissue image 102 can be acquired by a medical imaging scanner108 using any imaging modality, e.g., the scanner 108 may be an opticalcoherence tomography (OCT) scanner, a magnetic resonance imaging (MRI)scanner, an X-ray scanner, a computed tomography (CT) scanner, anultrasound (US) scanner, or a photographic scanner. The tissue image 102may include multiple images depicting the tissue in the region of thebody of the patient, e.g., that may be captured using different imagingmodalities. The region of the body of the patient 104 depicted by thetissue image 102 may be, for example, an eye, a breast, a prostate, abrain, or the whole body.

The medical condition may be, e.g., age-related macular degeneration(AMD—i.e., affecting eye tissue), a form of cancer (e.g., breast cancer,prostate cancer, or liver cancer), Alzheimer's, dementia, Parkinson'sdisease, chronic kidney disease (CKD), or any other medical condition.

The medical condition may be associated with a predefined set ofpossible states, where the state of the medical condition maycharacterize any aspect of how the medical condition is affecting thepatient. For example, the states of AMD may include: (i) an early, mildform of the condition, and (ii) a sight-threatening late form known asneovascular AMD (nAMD) or exudative wet AMD (exAMD). Optionally, the setof possible states of the medical condition may be understood to includea default “healthy” state, i.e., where the medical condition is notaffecting the patient.

In a particular example, the tissue image 102 may be an OCT imagedepicting the tissue in an eye of the patient 104, and the finalprogression score 106 may characterize a likelihood that the eye tissuewill progress from the early, mild form of AMD to sight-threatening nAMD(exAMD).

The system 100 may generate the final progression scores 106 using asegmentation engine 110, one or more progression classification neuralnetworks 112-A (that process tissue images), one or more progressionclassification neural networks 112-B (that process segmentation maps oftissue images), and an ensemble engine 114, each of which will bedescribed in more detail next.

The segmentation engine 110 is configured to process the tissue image102 to generate a corresponding segmentation map 116, i.e., thatsegments the tissue image 102 into a predefined set of possible tissueclasses. The segmentation map 116 may be a “hard” segmentation map or a“soft” segmentation map. A hard segmentation map associates each voxelof the tissue image 102 with a respective tissue class, while a softsegmentation map associates each voxel of the tissue image 102 with arespective probability of being included in each possible tissue class.

The segmentation map 116 may be represented, e.g., by a collection of“channels” including a respective channel corresponding to each possibletissue class, where the channel corresponding to a tissue class may berepresented as an ordered collection of numerical values having the samesize (dimensionality) as the tissue image 102. For a hard segmentationmap, the value of each voxel of a channel may define whether thecorresponding voxel of the tissue image 102 is included in the tissueclass corresponding to the channel. For a soft segmentation map, thevalue of each voxel of a channel may define the probability that thecorresponding voxel of the tissue image 102 is included in the tissueclass corresponding to the channel.

The set of possible tissue classes may include, e.g., anatomical tissueclasses (e.g., corresponding to healthy tissues), pathological tissueclasses (e.g., corresponding to diseased tissues), and artifacts (e.g.,distortions of the tissue image).

In a particular example, the tissue image 102 may be an OCT imagedepicting the tissue in an eye of the patient, and the segmentation map116 may segment the tissue image into a set of possible tissue classesincluding one or more of: vitreous and subhyaloid space (i.e., the areaabove the internal limiting membrane not covered by other tissueclasses), posterior hyaloid (i.e., the hyper-reflective membrane visibleabove the retina in cases of posterior vitreous detachment), epiretinalmembrane (i.e., the hyper-reflective band seen on the inner surface ofthe retina), neurosensory retina (i.e., all layers and contents of theretina excepting certain pathological features), intraretinal fluid(i.e., areas of round or oval hyporeflectivity located within theneurosensory retina), subretinal fluid (i.e., hyporeflective areas inthe subretinal space), subretinal hyperreflective material (i.e., areasof hyperreflectivity between the retinal and retinal pigment epithelium(RPE)), RPE (i.e., hyperreflective band underlying the neurosensoryretina), drusenoid pigment epithelium detachment (i.e., PED—elevation ofthe RPE and without the presence of fibrovascular material), serous PED(i.e., dome-shaped elevation of the RPE relative to Bruch's membrane),fibrovascular PED (i.e., irregular elevations of the RPE relative toBruch's membrane containing fibrovascular tissue), choroid and outerlayers (i.e., area below the RPE not covered by other tissue classes),mirror artifact (i.e., artefact caused by patient anatomy out of the OCTframe being reflected back onto the OCT), clipping artifact (i.e.,padding voxels introduced at the edges of the OCT slice during imageprocessing), and blink artifact (i.e., absent information due to patientblink), amongst others. In some implementations the tissue classesinclude a hyper-reflective foci (HRF) tissue class, wherehyper-reflective foci comprise well-circumscribed, dot- or oval-shapedlesions that are present within the intraretinal layers.

The segmentation engine 110 may generate the segmentation map 116 usinga set of segmentation neural networks, where each segmentation neuralnetwork is configured to process a tissue image to generate acorresponding segmentation map that segments the tissue image into thepredefined set of possible tissue classes. Each segmentation neuralnetwork may have been trained on a different set of training data, withdifferently initialized parameters, or both, from each othersegmentation neural network. To generate the segmentation map 116, thesegmentation engine 110 may process the tissue image 102 using eachsegmentation neural network to generate a respective “initial”segmentation map, and then combine (e.g., average) the initialsegmentation maps to generate the segmentation map 116. Generating thesegmentation map 116 using multiple segmentation neural networks mayincrease the accuracy and robustness of the segmentation map 116, e.g.,because a misclassification error by one segmentation neural network fora particular voxel may be corrected by other segmentation neuralnetworks that correctly classify the voxel.

In some implementations, rather than using segmentation neural networks,the segmentation engine 110 may generate the segmentation map 116 usingother types of machine learning models, e.g., random forests or supportvector machines (SVMs). Alternatively, rather than generating thesegmentation map 116 using one or more machine learning models, thesegmentation map 116 may be manually generated by a human expert (e.g.,a physician) and provided to the system 100 as a input, i.e., along withthe tissue image 102.

Each progression classification neural network 112-A is configured toprocess the tissue image 102 to generate a respective progression score118-A corresponding to each of the one or more future intervals of time.The system 100 may include multiple progression classification neuralnetworks 112-A, where each progression classification neural network112-A may have been trained on a different set of training data, withdifferently initialized parameters, or both, from each other progressionclassification neural network 112-A. Therefore, each progressionclassification neural network 112-A may potentially generate a differentprogression score 118-A by processing the same tissue image 102.

Each progression classification neural network 112-B is configured toprocess the segmentation map 116 corresponding to the tissue image 102to generate a respective progression score 118-B corresponding to eachof the one or more future intervals of time. Each progressionclassification neural network 112-B may have been trained on a differentset of training data, with differently initialized parameters, or both,from each other progression classification neural network 112-B.Therefore, each progression classification neural network 112-B maypotentially generate a different progression score 118-B by processingthe same segmentation map 116.

Optionally, the system 100 may generate additional progression scores118-A and 118-B using test-time data augmentation i.e. data augmentationwhen the system is used in inference. More specifically, the system 100may generate multiple respective versions of the tissue image 102 andthe segmentation map 116 by applying transformation operations (e.g.,random 3-D affine and elastic transformations) to the tissue image 102and the segmentation map 116. The system 100 may process each version ofthe tissue image 102 using each progression classification neuralnetwork 112-A to generate respective progression scores 118-A, and thesystem 100 may process each version of the segmentation map 116 usingeach progression classification neural network 112-B to generaterespective progression scores 118-B.

The ensemble engine 114 is configured to generate a respective finalprogression score 106 for each future interval of time by combining eachof the progression scores 118-A and 118-B for the future interval oftime. The ensemble engine 114 may combine the progression scores 118-Aand 118-B for a future interval of time, e.g., by computing the averageor the median of the progression scores 118-A and 118-B for the futureinterval of time.

The progression scores 118-A generated based on the tissue image 102 andthe progression scores 118-B generated based on the segmentation map 116may be different and complementary. For example, the progression scores118-A generated based on the tissue image 102 may reflect image featuresnot captured by the segmentation map 116 that may be relevant to theprogression prediction task, e.g., patterns in the reflectivity of thetissue in an OCT image of an eye. The progression scores 118-B generatedbased on the segmentation map 116 may be more stable (i.e., less likelyto assume inaccurate outlier values) than those generated using thetissue image 102, e.g., because of the potentially lower complexity ofthe segmentation map 116 relative to the tissue image 102. Therefore,generating the final progression scores 106 based on both the tissueimage 102 and the segmentation map 116 may improve the performance,e.g., the accuracy and robustness, of the system 100.

The system 100 may provide the segmentation map 116 of the tissue image102 to a user of the system 100 along with the progression scores forthe medical condition. The segmentation map 116 may enable the user tobetter understand and interpret the rationale used by the system 100 togenerate the progress scores, and thereby increase the confidence theuser may place in the progression scores.

The final progression scores 106 generated by the system 100 may beused, e.g., as part of a clinical workflow to facilitate decision-makingregarding whether and how to treat the patient 104 for the medicalcondition. For example, the final progression score 106 for a futureinterval of time (e.g., the next 6 months) being above a “treatmentthreshold” may be a relevant factor used to determine that preventativetreatments should be administered to the patient, or that regularfollow-ups should be performed to monitor the progression of the medicalcondition.

The treatment threshold may be determined, e.g., by using the system 100to generate a respective final progression score 106 for a future timeinterval for each tissue image in a set of validation data. A tissueimage may be included in the validation data if: (i) the progressionneural networks were not trained on the tissue image, and (ii) it isknown whether the state of the medical condition corresponding to thetissue image progressed to the target state in the future time interval.A treatment threshold may be selected from a range of possible treatmentthresholds, e.g., to achieve a desired sensitivity and or specificitysuch as 90% specificity or 80% sensitivity. Sensitivity may refer to aratio of: (i) the number of tissue images having a progression scoreabove the treatment threshold and for which the state of the medicalcondition progressed to the target state in the future time interval,and (ii) the number of tissue images for which the state of the medicalcondition progressed to the target state in the future time interval.Specificity may refer to a ratio of: (i) the number of tissue imageshaving a progression score below the treatment threshold and for whichthe state of the medical condition did not progress to the target statein the future time interval, and (ii) the number of tissue images forwhich the state of the medical condition did not progress to the targetstate in the future time interval.

Generally, the progression classification neural networks 112-A and112-B, and the segmentation neural networks used by the segmentationengine 110, may have any appropriate neural network architecture thatenables them to perform their described functions. For example, theirrespective architectures may include convolutional neural networklayers, pooling neural network layers, fully-connected neural networklayers, or a combination thereof, connected in any appropriateconfiguration. In one example, each progression classification neuralnetwork may include a sequence of “blocks”, where each block includes asequence of convolutional neural network layers having 3-D convolutionalkernels with dimensionality 1×3×3 or 3×1×1, and where the output of theblock includes a concatenation of the output of each convolutionalneural network layer in the block. The last convolutional block may befollowed by a fully-connected layer that outputs a respectiveprogression score corresponding to each of one or more future timeintervals. In another example, each segmentation neural network may havea 3-D U-Net neural network architecture.

The progression classification neural networks 112-A and 112-B and thesegmentation neural networks may be trained on respective sets oftraining data to optimize an objective function (e.g., a cross-entropyobjective function) using machine learning training techniques, e.g.,stochastic gradient descent. For example, the progression classificationneural networks may be trained on training examples that each include:(i) a training input, e.g., a training tissue image or a trainingsegmentation map, and (ii) a respective target progression score foreach of one or more future time intervals. The target progression scorefor a future time interval may be defined as the progression score thatshould be generated for the future time interval by a progression neuralnetwork by processing the training input. The target progression scorefor a future time interval may be represented, e.g., as a binary score0/1 indicating if the medical condition progressed to the target stateduring the future time interval. As another example, the segmentationneural networks may be trained on training examples that each include:(i) a training tissue image, and (ii) a target segmentation map thatshould be generated by the segmentation neural networks by processingthe training tissue image, e.g. derived from expert labelling.

Optionally, the progression classification neural networks may betrained to perform one or more “auxiliary” prediction tasks, i.e., bygenerating additional prediction outputs other than the progressionscores. Generally, training the progression classification neuralnetworks to perform auxiliary tasks may enable the progressionclassification neural networks to generate more effective internalrepresentations of network inputs, and thereby achieve improvedperformance, e.g., prediction accuracy. A few examples of auxiliaryprediction tasks are described in more detail next.

In one example, each progression classification neural network maygenerate a respective diagnosis score for each of one or more possiblemedical conditions that may be affecting the patient, where thediagnosis score for a possible medical condition characterizes alikelihood that the patient has the medical condition. Examples ofpossible medical conditions affecting the eye of a patient may include,e.g., macular retinal edema (MRO), choroidal neovascularization (CNV),and geographic atrophy.

In another example, each progression classification neural network maygenerate a referral score for each of multiple possible clinicalreferral decisions, where a clinical referral decision specifies anurgency with which the patient should receive further medical attention(e.g., by a specialist physician). The referral score for a referraldecision may represent a predicted likelihood that the referral decisionis the most appropriate referral decision for the patient. Examples ofreferral decision include: observation only, routine, semi-urgent, andurgent.

The progression classification neural networks may be trained to performthe auxiliary prediction tasks on a set of training data to optimize anobjective function (e.g., a cross-entropy objective function) usingmachine learning training techniques, e.g., stochastic gradient descent.For example, the progression classification neural networks may betrained on training examples that each include: (i) a training input,e.g., a training tissue image or a training segmentation map, and (ii)target auxiliary scores, e.g., diagnosis or referral scores, that shouldbe generated by the progression neural network by processing thetraining input. In some implementations, the system 100 may train theprogression classification neural networks to perform auxiliary tasksusing distillation training techniques. For example, the system 100 maygenerate the target auxiliary scores for training the progression neuralnetworks based on the outputs of another neural network that is trainedsolely to perform the auxiliary prediction task.

The system 100 described in this specification is widely applicable, andcan be applied to process various types of tissue images to generateprogression scores for various medical conditions. In a particularexample, the system 100 may be used to process an OCT image of an eye ofa patient to generate a progression score characterizing a likelihoodthat the eye will progress from the early, mild or dry form of AMD (orfrom a healthy state) to sight-threatening nAMD (exAMD) within a 6-monthperiod. Sight in an eye may be rapidly lost once nAMD develops andtreatment of nAMD is most effective if administered soon afterconversion, making the point of conversion from early AMD to nAMD (e.g.the treatment threshold) a critical moment in management of thisdisease. The system 100 can be used to identify patients with asignificant likelihood of developing nAMD in an eye within a 6-monthperiod, and these patients may have their eyes examined in regularfollow-ups to diagnose and treat nAMD soon after it develops.

FIG. 2 illustrates predictions made by the system 100 by processingmultiple OCT images of an eye of a patient over a period of months togenerate progression scores characterizing a likelihood that the eyewould progress from early or dry AMD to nAMD within a 6-month period.The horizontal axis 202 represents time (in months), the vertical axis204 represents progression score values, and each circle 206-A-Grepresents a final progression score generated by the system 100 byprocessing an OCT image captured at the corresponding time point. Thevertical line 208 represents the point of conversion of eye from earlyAMD to nAMD, and the horizontal lines 210-A-B represent possibletreatment thresholds, i.e., such that a progression score above thetreatment threshold indicates that nAMD is predicted to develop within a6-month period. It can be appreciated that, using either treatmentthreshold, progression scores generated by the system 100 would enableaccurate early prediction of the eventual conversion of the eye to nAMD,thereby enabling treatment to be commenced soon after conversion.

FIG. 3 is a flow diagram of an example process 300 for generating afinal progression score characterizing a likelihood that the state of amedical condition affecting a patient will progress to a target state ina future interval of time. For convenience, the process 300 will bedescribed as being performed by a system of one or more computerslocated in one or more locations. For example, a disease progressionprediction system, e.g., the disease progression prediction system 100of FIG. 1, appropriately programmed in accordance with thisspecification, can perform the process 300.

The system obtains an input image of tissue (e.g., eye tissue) capturedusing an imaging modality (302).

The system obtains a segmentation map of the tissue in the input imageinto multiple tissue types (304).

The system provides the input image to each of one or more firstprogression classification neural networks to obtain a respective firstprogression score from each first progression classification neuralnetwork (306). Each first progression classification neural network isconfigured to process an image of tissue captured using an imagingmodality to generate a first progression score characterizing alikelihood that a state of a medical condition affecting the tissue willprogress to a target state in the future interval of time.

The system provides the segmentation map to each of one or more secondprogression classification neural networks to obtain a respective secondprogression score from each second progression classification neuralnetwork (308). Each second progression classification neural network isconfigured to process a segmentation map of an image of tissue thatsegments the tissue in the image into multiple tissue types to generatea second progression score. Each second progression score characterizesa respective likelihood that a state of a medical condition affectingthe tissue will progress to a target state in the future interval oftime.

The system generates the final progression score based on the firstprogression scores and the second progression scores (310) e.g. byaveraging the first progression scores and the second progression scoresor by combining these in some other way.

This specification uses the term “configured” in connection with systemsand computer program components. For a system of one or more computersto be configured to perform particular operations or actions means thatthe system has installed on it software, firmware, hardware, or acombination of them that in operation cause the system to perform theoperations or actions. For one or more computer programs to beconfigured to perform particular operations or actions means that theone or more programs include instructions that, when executed by dataprocessing apparatus, cause the apparatus to perform the operations oractions.

Embodiments of the subject matter and the functional operationsdescribed in this specification can be implemented in digital electroniccircuitry, in tangibly-embodied computer software or firmware, incomputer hardware, including the structures disclosed in thisspecification and their structural equivalents, or in combinations ofone or more of them. Embodiments of the subject matter described in thisspecification can be implemented as one or more computer programs, i.e.,one or more modules of computer program instructions encoded on atangible non-transitory storage medium for execution by, or to controlthe operation of, data processing apparatus. The computer storage mediumcan be a machine-readable storage device, a machine-readable storagesubstrate, a random or serial access memory device, or a combination ofone or more of them. Alternatively or in addition, the programinstructions can be encoded on an artificially-generated propagatedsignal, e.g., a machine-generated electrical, optical, orelectromagnetic signal, that is generated to encode information fortransmission to suitable receiver apparatus for execution by a dataprocessing apparatus.

The term “data processing apparatus” refers to data processing hardwareand encompasses all kinds of apparatus, devices, and machines forprocessing data, including by way of example a programmable processor, acomputer, or multiple processors or computers. The apparatus can alsobe, or further include, special purpose logic circuitry, e.g., an FPGA(field programmable gate array) or an ASIC (application-specificintegrated circuit). The apparatus can optionally include, in additionto hardware, code that creates an execution environment for computerprograms, e.g., code that constitutes processor firmware, a protocolstack, a database management system, an operating system, or acombination of one or more of them.

A computer program, which may also be referred to or described as aprogram, software, a software application, an app, a module, a softwaremodule, a script, or code, can be written in any form of programminglanguage, including compiled or interpreted languages, or declarative orprocedural languages; and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, or other unitsuitable for use in a computing environment. A program may, but neednot, correspond to a file in a file system. A program can be stored in aportion of a file that holds other programs or data, e.g., one or morescripts stored in a markup language document, in a single file dedicatedto the program in question, or in multiple coordinated files, e.g.,files that store one or more modules, sub-programs, or portions of code.A computer program can be deployed to be executed on one computer or onmultiple computers that are located at one site or distributed acrossmultiple sites and interconnected by a data communication network.

In this specification the term “engine” is used broadly to refer to asoftware-based system, subsystem, or process that is programmed toperform one or more specific functions. Generally, an engine will beimplemented as one or more software modules or components, installed onone or more computers in one or more locations. In some cases, one ormore computers will be dedicated to a particular engine; in other cases,multiple engines can be installed and running on the same computer orcomputers.

The processes and logic flows described in this specification can beperformed by one or more programmable computers executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby special purpose logic circuitry, e.g., an FPGA or an ASIC, or by acombination of special purpose logic circuitry and one or moreprogrammed computers.

Computers suitable for the execution of a computer program can be basedon general or special purpose microprocessors or both, or any other kindof central processing unit. Generally, a central processing unit willreceive instructions and data from a read-only memory or a random accessmemory or both. The essential elements of a computer are a centralprocessing unit for performing or executing instructions and one or morememory devices for storing instructions and data. The central processingunit and the memory can be supplemented by, or incorporated in, specialpurpose logic circuitry. Generally, a computer will also include, or beoperatively coupled to receive data from or transfer data to, or both,one or more mass storage devices for storing data, e.g., magnetic,magneto-optical disks, or optical disks. However, a computer need nothave such devices. Moreover, a computer can be embedded in anotherdevice, e.g., a mobile telephone, a personal digital assistant (PDA), amobile audio or video player, a game console, a Global PositioningSystem (GPS) receiver, or a portable storage device, e.g., a universalserial bus (USB) flash drive, to name just a few.

Computer-readable media suitable for storing computer programinstructions and data include all forms of non-volatile memory, mediaand memory devices, including by way of example semiconductor memorydevices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks,e.g., internal hard disks or removable disks; magneto-optical disks; andCD-ROM and DVD-ROM disks.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device that is usedby the user; for example, by sending web pages to a web browser on auser's device in response to requests received from the web browser.Also, a computer can interact with a user by sending text messages orother forms of message to a personal device, e.g., a smartphone that isrunning a messaging application, and receiving responsive messages fromthe user in return.

Data processing apparatus for implementing machine learning models canalso include, for example, special-purpose hardware accelerator unitsfor processing common and compute-intensive parts of machine learningtraining or production, i.e., inference, workloads.

Machine learning models can be implemented and deployed using a machinelearning framework, e.g., a TensorFlow framework, a Microsoft CognitiveToolkit framework, an Apache Singa framework, or an Apache MXNetframework.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back-end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front-end component, e.g., aclient computer having a graphical user interface, a web browser, or anapp through which a user can interact with an implementation of thesubject matter described in this specification, or any combination ofone or more such back-end, middleware, or front-end components. Thecomponents of the system can be interconnected by any form or medium ofdigital data communication, e.g., a communication network. Examples ofcommunication networks include a local area network (LAN) and a widearea network (WAN), e.g., the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. In someembodiments, a server transmits data, e.g., an HTML page, to a userdevice, e.g., for purposes of displaying data to and receiving userinput from a user interacting with the device, which acts as a client.Data generated at the user device, e.g., a result of the userinteraction, can be received at the server from the device.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinvention or on the scope of what may be claimed, but rather asdescriptions of features that may be specific to particular embodimentsof particular inventions. Certain features that are described in thisspecification in the context of separate embodiments can also beimplemented in combination in a single embodiment. Conversely, variousfeatures that are described in the context of a single embodiment canalso be implemented in multiple embodiments separately or in anysuitable subcombination. Moreover, although features may be describedabove as acting in certain combinations and even initially be claimed assuch, one or more features from a claimed combination can in some casesbe excised from the combination, and the claimed combination may bedirected to a subcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings and recited inthe claims in a particular order, this should not be understood asrequiring that such operations be performed in the particular ordershown or in sequential order, or that all illustrated operations beperformed, to achieve desirable results. In certain circumstances,multitasking and parallel processing may be advantageous. Moreover, theseparation of various system modules and components in the embodimentsdescribed above should not be understood as requiring such separation inall embodiments, and it should be understood that the described programcomponents and systems can generally be integrated together in a singlesoftware product or packaged into multiple software products.

Particular embodiments of the subject matter have been described. Otherembodiments are within the scope of the following claims. For example,the actions recited in the claims can be performed in a different orderand still achieve desirable results. As one example, the processesdepicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirableresults. In some cases, multitasking and parallel processing may beadvantageous.

What is claimed is:
 1. A system comprising one or more computers and oneor more storage devices storing instructions that when executed by theone or more computers cause the one or more computers to implement: oneor more first classification neural networks, wherein each firstclassification neural network is configured to: receive an image of eyetissue captured using an imaging modality; and process the image togenerate a first progression score characterizing a likelihood that astate of a medical condition affecting the eye tissue will progress to atarget state in a future interval of time; one or more secondclassification neural networks, wherein each second classificationneural network is configured to: receive a segmentation map of an imageof eye tissue that segments the eye tissue in the image into a pluralityof tissue types; and process the segmentation map to generate a secondprogression score characterizing a likelihood that a state of a medicalcondition affecting the eye tissue will progress to a target state in afuture interval of time; a subsystem configured to: obtain: (i) an inputimage of eye tissue captured using an imaging modality, and (ii) asegmentation map of the eye tissue in the input image into a pluralityof tissue types; and generate, based on the input image and thesegmentation map, a final progression score characterizing a likelihoodthat a state of a medical condition affecting the eye tissue willprogress to a target state in a future interval of time, comprising:providing the input image to each of the first classification neuralnetworks to obtain a respective first progression score from each firstclassification neural network; providing the segmentation map to each ofthe second classification neural networks to obtain a respective secondprogression score from each second classification neural network; andgenerating the final progression score based on the first progressionscores and the second progression scores.
 2. The system of claim 1,wherein the imaging modality is an optical coherence tomography (OCT)modality.
 3. The system of claim 1, wherein the medical conditionaffecting the eye tissue is age-related macular degeneration (AMD). 4.The system of claim 3, wherein the target state of the medical conditionaffecting the eye tissue is neovascular age-related macular degeneration(nAMD).
 5. The system of claim 1, further comprising: one or moresegmentation neural networks, wherein each segmentation neural networkis configured to: receive an image of eye tissue captured using theimaging modality; and process the image to generate a segmentation mapof the image that segments the eye tissue in the image into a pluralityof tissue types; wherein the subsystem is further configured to: providethe input image to each of the segmentation neural networks to obtain arespective initial segmentation map of the eye tissue in the input imageinto the plurality of tissue types from each segmentation neuralnetwork; and generate the segmentation map based on the initialsegmentation maps.
 6. The system of claim 5, wherein generating thesegmentation map based on the initial segmentation maps comprisesaveraging a plurality of the initial segmentation maps.
 7. The system ofclaim 5, wherein each segmentation neural network is a convolutionalneural network having a U-Net architecture.
 8. The system of claim 1,wherein the input image of eye tissue captured using the imagingmodality is a three-dimensional image comprising a plurality of voxels,and wherein the segmentation map assigns a respective tissue type from apredetermined set of tissue types to each of the voxels.
 9. The systemof claim 8, wherein the predetermined set of tissue types comprise oneor more anatomical tissue types and one or more pathological tissuetypes.
 10. The system of claim 1, wherein the future interval of time isan interval of time starting from a current time point.
 11. The systemof claim 1, further comprising generating, for each of a plurality ofgiven future intervals of time, a respective final progression scorecharacterizing a likelihood that the state of the medical conditionaffecting the eye tissue will progress to the target state in the givenfuture interval of time.
 12. The system of claim 1, further comprisingdetermining, based on the final progression score, that preventativetreatment should be administered to the eye tissue.
 13. The system ofclaim 1, wherein the first classification neural networks and the secondclassification neural networks are trained to generate additionaloutputs characterizing referral decisions and additional diagnoses. 14.The system of claim 1, wherein: the system comprises a plurality offirst classification neural networks, wherein each first classificationneural network has a same architecture but has been trained (i) on adifferent set of training data, (ii) with differently initializedparameters, or (iii) both, from each other first classification neuralnetwork.
 15. The system of claim 1, wherein: the system comprises aplurality of second classification neural networks, wherein each secondclassification neural network has a same architecture but has beentrained (i) on a different set of training data, (ii) with differentlyinitialized parameters, or (iii) both, from each other secondclassification neural network.
 16. The system of claim 1, whereinproviding the input image to each of the first classification neuralnetworks and providing the segmentation map to each of the secondclassification neural networks comprises performing test-time dataaugmentation.
 17. The system of claim 1, wherein generating the finalprogression score based on the first progression scores and the secondprogression scores comprises averaging the first progression scores andthe second progression scores.
 18. (canceled)
 19. (canceled)
 20. Amethod performed by one or more data processing apparatus, the methodcomprising: obtaining: (i) an input image of eye tissue captured usingan imaging modality, and (ii) a segmentation map of the eye tissue inthe input image into a plurality of tissue types; and generating, basedon the input image and the segmentation map, a final progression scorecharacterizing a likelihood that a state of a medical conditionaffecting the eye tissue will progress to a target state in a futureinterval of time, comprising: providing the input image to each of oneor more first classification neural networks to obtain a respectivefirst progression score from each first classification neural network,wherein each first classification neural network is configured to:receive the input image of eye tissue captured using the imagingmodality; and process the input image to generate a first progressionscore characterizing a likelihood that a state of a medical conditionaffecting the eye tissue will progress to the target state in the futureinterval of time; providing the segmentation map to each of one or moresecond classification neural networks to obtain a respective secondprogression score from each second classification neural network,wherein each second classification neural network is configured to:receive the segmentation map of the eye tissue in the input image intothe plurality of tissue types; and process the segmentation map togenerate a second progression score characterizing a likelihood that astate of a medical condition affecting the eye tissue will progress tothe target state in the future interval of time; and generating thefinal progression score based on the first progression scores and thesecond progression scores.
 21. The method of claim 20, wherein theimaging modality is an optical coherence tomography (OCT) modality. 22.One or more non-transitory computer storage media storing instructionsthat when executed by one or more computers cause the one or morecomputers to perform operations comprising: obtaining: (i) an inputimage of eye tissue captured using an imaging modality, and (ii) asegmentation map of the eye tissue in the input image into a pluralityof tissue types; and generating, based on the input image and thesegmentation map, a final progression score characterizing a likelihoodthat a state of a medical condition affecting the eye tissue willprogress to a target state in a future interval of time, comprising:providing the input image to each of one or more first classificationneural networks to obtain a respective first progression score from eachfirst classification neural network, wherein each first classificationneural network is configured to: receive the input image of eye tissuecaptured using the imaging modality; and process the input image togenerate a first progression score characterizing a likelihood that astate of a medical condition affecting the eye tissue will progress tothe target state in the future interval of time; providing thesegmentation map to each of one or more second classification neuralnetworks to obtain a respective second progression score from eachsecond classification neural network, wherein each second classificationneural network is configured to: receive the segmentation map of the eyetissue in the input image into the plurality of tissue types; andprocess the segmentation map to generate a second progression scorecharacterizing a likelihood that a state of a medical conditionaffecting the eye tissue will progress to the target state in the futureinterval of time; and generating the final progression score based onthe first progression scores and the second progression scores.